Skip to content

Fix TOCTOU crash in rotate_log() under concurrent invocations - #136

Merged
ehelms merged 2 commits into
theforeman:masterfrom
pablomh:fix/rotate-log-toctou
Aug 19, 2026
Merged

Fix TOCTOU crash in rotate_log() under concurrent invocations#136
ehelms merged 2 commits into
theforeman:masterfrom
pablomh:fix/rotate-log-toctou

Conversation

@pablomh

@pablomh pablomh commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Summary

rotate_log() checks os.path.exists(log_path) and then calls os.rename(log_path, backup_path), with no exclusion between the two steps. When multiple obsah-based CLIs (e.g. foremanctl) run concurrently and share the same log file — a normal situation, since log rotation happens unconditionally at the top of main() before any other work — one process can rename the file away in the gap between another process's exists() check and its own rename() call, crashing with an unhandled FileNotFoundError.

Live reproduction

Confirmed against the real installed package (obsah 1.10.0, unmodified) on a live Satellite host, using foremanctl auth-bundle (which shares one log file across all invocations): 150 concurrent invocations across several rounds crashed 8 of them with this exact traceback:

File ".../obsah/__init__.py", line 596, in main
    rotate_log(application_config.log_path())
  File ".../obsah/__init__.py", line 585, in rotate_log
    os.rename(log_path, backup_path)
FileNotFoundError: [Errno 2] No such file or directory: '/var/log/foremanctl/foremanctl.log' -> '/var/log/foremanctl/foremanctl.2026-08-18-07-46-18.log'

After applying this fix on the same host, 120 further concurrent invocations produced zero crashes.

Why no locking is needed here

Unlike a genuine read-modify-write (e.g. merging into a shared parameters file), log rotation doesn't need mutual exclusion. Every concurrent process wants the same outcome — "the old log is archived and a fresh one starts" — and that outcome is fully satisfied once any one process's rename succeeds. There's no per-process contribution to preserve, so the fix is simply to tolerate losing the race rather than to coordinate around it.

This is a well-established idiom rather than something new to this codebase:

  • It's the textbook EAFP ("easier to ask forgiveness than permission") pattern the Python community favors over check-then-act for exactly this reason — the original code was LBYL ("look before you leap"), which is inherently racy against other processes.
  • persist_args(), a few lines away in this same file, already uses os.makedirs(persist_dir, mode=0o770, exist_ok=True) to solve the identical TOCTOU shape for directory creation (exist_ok=True exists in the stdlib specifically for this). This PR applies the same idiom to os.rename().
  • Ansible itself does the same thing for an analogous shared-resource cleanup race in lib/ansible/plugins/connection/ssh.py:
    with contextlib.suppress(FileNotFoundError):
        self.shm.unlink()

Testing

Added test_rotate_log_tolerates_concurrent_rotation, which reconstructs the actual race (not just a raised exception): os.path.exists() is patched so that, on the one call that matters, it returns True but also renames the file away as a side effect — simulating another process winning the race — so the real, unmocked os.rename() call fails naturally. Asserts rotate_log() doesn't raise, that the "stolen" file exists (confirming the simulated race actually happened), and that the original log file is gone (matching the real post-race state).

Co-Authored-By: Claude Sonnet 5 noreply@anthropic.com

rotate_log() checks os.path.exists(log_path) then calls os.rename(),
with no exclusion between the two. When multiple obsah invocations
share the same log file (the common case), one process can rename the
file away between another's exists() check and its own rename(),
crashing with an unhandled FileNotFoundError.

Confirmed live against the real installed package (obsah 1.10.0, not
this patched fork): 150 concurrent `foremanctl auth-bundle` invocations
against a shared log file crashed 8 of them this way across several
rounds. After suppressing FileNotFoundError around the rename, 120
further concurrent invocations produced zero crashes.

Unlike the parameters.yaml read-modify-write fixed in the previous two
commits, this doesn't need mutual exclusion: it doesn't matter which
process wins the rotation, only that the loser doesn't crash, so a
plain contextlib.suppress is sufficient.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Comment thread tests/test_log_rotate.py Outdated

def test_rotate_log_tolerates_concurrent_rotation(tmp_path):
# Another concurrent obsah invocation can rename the same log file away
# between our exists() check and our own rename.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not seeing how to connect this comment to the test itself. Maybe drop it?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair point — that comment was describing the production race, not this test. The test never starts another process; it patches exists() so the file is already gone when rename() runs. Dropped it.

The test simulates the race by stealing the file from a patched exists()
check, not by starting another process, so the old comment was misleading.

Co-authored-by: Cursor <cursoragent@cursor.com>
@ehelms
ehelms merged commit 98a5988 into theforeman:master Aug 19, 2026
10 checks passed
@pablomh
pablomh deleted the fix/rotate-log-toctou branch August 19, 2026 17:16
pablomh added a commit to redhat-performance/satperf that referenced this pull request Aug 19, 2026
…e to auth-bundle

Bundle of independent foremanctl role improvements, tracked separately
from load-balanced capsule support (not committed here - unconfirmed,
stays local until proven working):

- Split foremanctl_features into foremanctl_add_features/
  foremanctl_remove_features, enabling selective feature removal (e.g.
  cloud-connector) rather than only addition.
- COPR cleanup: de-nest the EL9 COPR task; EL10 now enables the
  official @theforeman/{foreman,katello,plugins}-nightly-staging COPRs
  via a loop instead of the old personal ekohl/foreman-nightly-staging
  COPR.
- Add a step to apply satellite vendor overrides before deployment.
- Deployment robustness: replace the old ignore_errors/failed_when:
  ... is failed debug-task pattern with failed_when: false plus an
  explicit ansible.builtin.assert on rc == 0, for both the initial
  deploy and the add/remove-features deploy.
- Redact the admin password in the initial-deploy debug output - a
  real secret-leak fix, since a failed task previously echoed the raw
  password via Ansible's own fatal-task output.
- Rename certificate-bundle to auth-bundle, tracking foremanctl's own
  CLI rename, and drop the old separate OAuth consumer key/secret file
  handling now that the auth bundle covers it. The new "Generate proxy
  auth bundle" task's throttle: 1 references theforeman/obsah#136 (a
  rotate_log() TOCTOU race under concurrent capsule delegation) - the
  same obsah concurrency class as the separately-planned parameters.yaml
  locking fix.
- Make foremanctl_deployment_type a required, explicit per-playbook
  variable instead of a role default: foremanctl_specific.yaml sets it
  to server, and the new foremanctl_capsules_specific.yaml sets it to
  proxy for capsule/smart-proxy hosts.
- Restructure proxy flavor selection into _foremanctl_proxy_flavor
  (Foreman -> foreman-proxy-content, Satellite -> capsule), replacing
  the static foremanctl_proxy_flavor default.
- capsule role: restructure certs-generate so Foreman/Satellite
  fact-setting happens inside the certs-generate block, and gate
  concurrent vs. sequential certs-generate execution on
  sat_version == 'stream' vs. not, replacing ad hoc throttle: 1
  "XXX: Submit PR" hacks on the installer-script and
  refresh-features tasks.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants